chore: clustering mode is required for pyroscope #47
+43
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Profiling cannot occur by 2 agents at the same time. Clustering distributed these jobs evenly across agents
Prometheus ignores this, as clustering block is not used
| @@ -210,6 +210,10 @@ pyroscope.scrape "pods" { | |||
|
|
|||
| scrape_interval = "{{ .Values.global.telemetry.profiling.scrape_interval }}" | |||
| scrape_timeout = "{{ .Values.global.telemetry.profiling.scrape_timeout }}" | |||
|
|
|||
| clustering { | |||
| enabled = true | |||
There was a problem hiding this comment.
Are we enabling this by default with no means to disable it intentionally?
There was a problem hiding this comment.
It is set to true. When grafana agents are not running in cluster mode, this block is a no-op:
https://grafana.com/docs/agent/latest/flow/reference/components/pyroscope.scrape/#clustering-block
If Grafana Agent Flow is not running in clustered mode, this block is a no-op.
So you can disable this behavior by doing:
grafana-agent:
agent:
clustering:
enabled: false # <- Now the clustering block in the agent config is a no-opThere was a problem hiding this comment.
So I guess the question is, is there a use case to run agents in cluster mode, but not use cluster mode for the profiling.
As of now, I don't think there is. If we use more clustering features, then maybe we can add more knobs.
dannykopping
approved these changes
Aug 18, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Profiling cannot occur by 2 agents at the same time. Clustering distributed these jobs evenly across agents.
This is to solve this error in production on agents:
Validated locally
Ran without clustering with 4 nodes and 1 coder, I get the error without fail.
Extra
I bet this prometheus error is a similar problem:
Maybe we should cluster scrape prometheus as well?